{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Benchmark of various outlier detection models\n",
    "\n",
    "### The models are evaluated by ROC, Precision @ n and execution time on 17 benchmark datasets. All datasets are split (60% for training and 40% for testing). The full result by averaging 10 indepent trials can be found [here](https://pyod.readthedocs.io/en/latest/benchmark.html).\n",
    "\n",
    "**[PyOD](https://github.com/yzhao062/pyod)** is a comprehensive **Python toolkit** to **identify outlying objects** in \n",
    "multivariate data with both unsupervised and supervised approaches.\n",
    "The model covered in this example includes:\n",
    "\n",
    "  1. Linear Models for Outlier Detection:\n",
    "     1. **PCA: Principal Component Analysis** use the sum of\n",
    "       weighted projected distances to the eigenvector hyperplane \n",
    "       as the outlier outlier scores)\n",
    "     2. **MCD: Minimum Covariance Determinant** (use the mahalanobis distances \n",
    "       as the outlier scores)\n",
    "     3. **OCSVM: One-Class Support Vector Machines**\n",
    "     \n",
    "  2. Proximity-Based Outlier Detection Models:\n",
    "     1. **LOF: Local Outlier Factor**\n",
    "     2. **CBLOF: Clustering-Based Local Outlier Factor**\n",
    "     3. **kNN: k Nearest Neighbors** (use the distance to the kth nearest \n",
    "     neighbor as the outlier score)\n",
    "     4. **HBOS: Histogram-based Outlier Score**\n",
    "     \n",
    "  3. Probabilistic Models for Outlier Detection:\n",
    "     1. **ABOD: Angle-Based Outlier Detection**\n",
    "  \n",
    "  4. Outlier Ensembles and Combination Frameworks\n",
    "     1. **Isolation Forest**\n",
    "     2. **Feature Bagging**\n",
    "\n",
    "     \n",
    "Corresponding file could be found at /examples/compare_all_models.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "\n",
    "import os\n",
    "import sys\n",
    "from time import time\n",
    "\n",
    "# temporary solution for relative imports in case pyod is not installed\n",
    "# if pyod is installed, no need to use the following line\n",
    "sys.path.append(\n",
    "    os.path.abspath(os.path.join(os.path.dirname(\"__file__\"), '..')))\n",
    "# supress warnings for clean output\n",
    "import warnings\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "from scipy.io import loadmat\n",
    "\n",
    "from pyod.models.abod import ABOD\n",
    "from pyod.models.cblof import CBLOF\n",
    "from pyod.models.feature_bagging import FeatureBagging\n",
    "from pyod.models.hbos import HBOS\n",
    "from pyod.models.iforest import IForest\n",
    "from pyod.models.knn import KNN\n",
    "from pyod.models.lof import LOF\n",
    "from pyod.models.mcd import MCD\n",
    "from pyod.models.ocsvm import OCSVM\n",
    "from pyod.models.pca import PCA\n",
    "\n",
    "from pyod.utils.utility import standardizer\n",
    "from pyod.utils.utility import precision_n_scores\n",
    "from sklearn.metrics import roc_auc_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "... Processing arrhythmia.mat ...\n",
      "Angle-based Outlier Detector (ABOD) ROC:0.7687, precision @ rank n:0.3571, execution time: 1.126s\n",
      "Cluster-based Local Outlier Factor ROC:0.7824, precision @ rank n:0.4643, execution time: 0.9786s\n",
      "Feature Bagging ROC:0.7796, precision @ rank n:0.4643, execution time: 0.6076s\n",
      "Histogram-base Outlier Detection (HBOS) ROC:0.8511, precision @ rank n:0.5714, execution time: 0.8914s\n",
      "Isolation Forest ROC:0.8639, precision @ rank n:0.6071, execution time: 0.2507s\n",
      "K Nearest Neighbors (KNN) ROC:0.782, precision @ rank n:0.5, execution time: 0.1133s\n",
      "Local Outlier Factor (LOF) ROC:0.7787, precision @ rank n:0.4643, execution time: 0.0742s\n",
      "Minimum Covariance Determinant (MCD) ROC:0.8228, precision @ rank n:0.4286, execution time: 1.4578s\n",
      "One-class SVM (OCSVM) ROC:0.7986, precision @ rank n:0.5, execution time: 0.0481s\n",
      "Principal Component Analysis (PCA) ROC:0.7997, precision @ rank n:0.5, execution time: 0.0642s\n",
      "\n",
      "... Processing cardio.mat ...\n",
      "Angle-based Outlier Detector (ABOD) ROC:0.5892, precision @ rank n:0.1918, execution time: 0.3579s\n",
      "Cluster-based Local Outlier Factor ROC:0.973, precision @ rank n:0.7945, execution time: 0.0983s\n",
      "Feature Bagging ROC:0.6385, precision @ rank n:0.1781, execution time: 0.8683s\n",
      "Histogram-base Outlier Detection (HBOS) ROC:0.8373, precision @ rank n:0.4521, execution time: 0.005s\n",
      "Isolation Forest ROC:0.9502, precision @ rank n:0.6027, execution time: 0.2767s\n",
      "K Nearest Neighbors (KNN) ROC:0.734, precision @ rank n:0.3562, execution time: 0.1905s\n",
      "Local Outlier Factor (LOF) ROC:0.588, precision @ rank n:0.1507, execution time: 0.0993s\n",
      "Minimum Covariance Determinant (MCD) ROC:0.8195, precision @ rank n:0.411, execution time: 0.5835s\n",
      "One-class SVM (OCSVM) ROC:0.9478, precision @ rank n:0.5342, execution time: 0.0822s\n",
      "Principal Component Analysis (PCA) ROC:0.9616, precision @ rank n:0.6849, execution time: 0.003s\n",
      "\n",
      "... Processing glass.mat ...\n",
      "Angle-based Outlier Detector (ABOD) ROC:0.6951, precision @ rank n:0.25, execution time: 0.0361s\n",
      "Cluster-based Local Outlier Factor ROC:0.7957, precision @ rank n:0.25, execution time: 0.0251s\n",
      "Feature Bagging ROC:0.7073, precision @ rank n:0.25, execution time: 0.0351s\n",
      "Histogram-base Outlier Detection (HBOS) ROC:0.7073, precision @ rank n:0.0, execution time: 0.005s\n",
      "Isolation Forest ROC:0.7134, precision @ rank n:0.25, execution time: 0.1584s\n",
      "K Nearest Neighbors (KNN) ROC:0.8384, precision @ rank n:0.25, execution time: 0.013s\n",
      "Local Outlier Factor (LOF) ROC:0.7043, precision @ rank n:0.25, execution time: 0.003s\n",
      "Minimum Covariance Determinant (MCD) ROC:0.8293, precision @ rank n:0.0, execution time: 0.0481s\n",
      "One-class SVM (OCSVM) ROC:0.6585, precision @ rank n:0.25, execution time: 0.001s\n",
      "Principal Component Analysis (PCA) ROC:0.686, precision @ rank n:0.25, execution time: 0.001s\n",
      "\n",
      "... Processing ionosphere.mat ...\n",
      "Angle-based Outlier Detector (ABOD) ROC:0.9181, precision @ rank n:0.8431, execution time: 0.0602s\n",
      "Cluster-based Local Outlier Factor ROC:0.795, precision @ rank n:0.549, execution time: 0.0381s\n",
      "Feature Bagging ROC:0.9303, precision @ rank n:0.8039, execution time: 0.0642s\n",
      "Histogram-base Outlier Detection (HBOS) ROC:0.6052, precision @ rank n:0.3922, execution time: 0.006s\n",
      "Isolation Forest ROC:0.8486, precision @ rank n:0.5882, execution time: 0.1785s\n",
      "K Nearest Neighbors (KNN) ROC:0.932, precision @ rank n:0.8824, execution time: 0.0251s\n",
      "Local Outlier Factor (LOF) ROC:0.9227, precision @ rank n:0.7843, execution time: 0.007s\n",
      "Minimum Covariance Determinant (MCD) ROC:0.9669, precision @ rank n:0.8627, execution time: 0.0622s\n",
      "One-class SVM (OCSVM) ROC:0.8257, precision @ rank n:0.6863, execution time: 0.004s\n",
      "Principal Component Analysis (PCA) ROC:0.7941, precision @ rank n:0.5686, execution time: 0.003s\n",
      "\n",
      "... Processing letter.mat ...\n",
      "Angle-based Outlier Detector (ABOD) ROC:0.8783, precision @ rank n:0.4375, execution time: 0.365s\n",
      "Cluster-based Local Outlier Factor ROC:0.5301, precision @ rank n:0.0312, execution time: 0.1052s\n",
      "Feature Bagging ROC:0.8947, precision @ rank n:0.4062, execution time: 0.7309s\n",
      "Histogram-base Outlier Detection (HBOS) ROC:0.6063, precision @ rank n:0.0938, execution time: 0.007s\n",
      "Isolation Forest ROC:0.6201, precision @ rank n:0.0625, execution time: 0.2416s\n",
      "K Nearest Neighbors (KNN) ROC:0.8573, precision @ rank n:0.3125, execution time: 0.1905s\n",
      "Local Outlier Factor (LOF) ROC:0.8765, precision @ rank n:0.3438, execution time: 0.0922s\n",
      "Minimum Covariance Determinant (MCD) ROC:0.8061, precision @ rank n:0.1875, execution time: 1.3225s\n",
      "One-class SVM (OCSVM) ROC:0.5927, precision @ rank n:0.125, execution time: 0.0973s\n",
      "Principal Component Analysis (PCA) ROC:0.5216, precision @ rank n:0.125, execution time: 0.005s\n",
      "\n",
      "... Processing lympho.mat ...\n",
      "Angle-based Outlier Detector (ABOD) ROC:0.9831, precision @ rank n:0.0, execution time: 0.0261s\n",
      "Cluster-based Local Outlier Factor ROC:1.0, precision @ rank n:1.0, execution time: 0.0321s\n",
      "Feature Bagging ROC:1.0, precision @ rank n:1.0, execution time: 0.0261s\n",
      "Histogram-base Outlier Detection (HBOS) ROC:1.0, precision @ rank n:1.0, execution time: 0.005s\n",
      "Isolation Forest ROC:1.0, precision @ rank n:1.0, execution time: 0.1795s\n",
      "K Nearest Neighbors (KNN) ROC:1.0, precision @ rank n:1.0, execution time: 0.011s\n",
      "Local Outlier Factor (LOF) ROC:1.0, precision @ rank n:1.0, execution time: 0.003s\n",
      "Minimum Covariance Determinant (MCD) ROC:1.0, precision @ rank n:1.0, execution time: 0.0531s\n",
      "One-class SVM (OCSVM) ROC:1.0, precision @ rank n:1.0, execution time: 0.002s\n",
      "Principal Component Analysis (PCA) ROC:1.0, precision @ rank n:1.0, execution time: 0.001s\n",
      "\n",
      "... Processing mnist.mat ...\n",
      "Angle-based Outlier Detector (ABOD) ROC:0.7628, precision @ rank n:0.3367, execution time: 6.8502s\n",
      "Cluster-based Local Outlier Factor ROC:0.8204, precision @ rank n:0.3605, execution time: 0.9004s\n",
      "Feature Bagging ROC:0.7157, precision @ rank n:0.3741, execution time: 44.9604s\n",
      "Histogram-base Outlier Detection (HBOS) ROC:0.5766, precision @ rank n:0.1361, execution time: 0.0411s\n",
      "Isolation Forest ROC:0.7939, precision @ rank n:0.2721, execution time: 1.6943s\n",
      "K Nearest Neighbors (KNN) ROC:0.8498, precision @ rank n:0.432, execution time: 6.6327s\n",
      "Local Outlier Factor (LOF) ROC:0.7195, precision @ rank n:0.3673, execution time: 5.9395s\n",
      "Minimum Covariance Determinant (MCD) ROC:0.8713, precision @ rank n:0.2653, execution time: 4.1711s\n",
      "One-class SVM (OCSVM) ROC:0.854, precision @ rank n:0.3946, execution time: 4.3626s\n",
      "Principal Component Analysis (PCA) ROC:0.8534, precision @ rank n:0.3878, execution time: 0.1433s\n",
      "\n",
      "... Processing musk.mat ...\n",
      "Angle-based Outlier Detector (ABOD) ROC:0.2161, precision @ rank n:0.1, execution time: 1.8951s\n",
      "Cluster-based Local Outlier Factor ROC:0.9899, precision @ rank n:0.65, execution time: 0.2998s\n",
      "Feature Bagging ROC:0.473, precision @ rank n:0.125, execution time: 12.2746s\n",
      "Histogram-base Outlier Detection (HBOS) ROC:0.9999, precision @ rank n:0.975, execution time: 0.0582s\n",
      "Isolation Forest ROC:1.0, precision @ rank n:1.0, execution time: 1.0528s\n",
      "K Nearest Neighbors (KNN) ROC:0.8009, precision @ rank n:0.175, execution time: 1.7376s\n",
      "Local Outlier Factor (LOF) ROC:0.4629, precision @ rank n:0.125, execution time: 1.5411s\n",
      "Minimum Covariance Determinant (MCD) ROC:1.0, precision @ rank n:1.0, execution time: 21.114s\n",
      "One-class SVM (OCSVM) ROC:1.0, precision @ rank n:1.0, execution time: 1.1922s\n",
      "Principal Component Analysis (PCA) ROC:1.0, precision @ rank n:1.0, execution time: 0.1504s\n",
      "\n",
      "... Processing optdigits.mat ...\n",
      "Angle-based Outlier Detector (ABOD) ROC:0.4894, precision @ rank n:0.0152, execution time: 2.243s\n",
      "Cluster-based Local Outlier Factor ROC:0.5329, precision @ rank n:0.0, execution time: 0.3911s\n",
      "Feature Bagging ROC:0.5062, precision @ rank n:0.0303, execution time: 11.9945s\n",
      "Histogram-base Outlier Detection (HBOS) ROC:0.8774, precision @ rank n:0.2121, execution time: 0.0271s\n",
      "Isolation Forest ROC:0.6735, precision @ rank n:0.0303, execution time: 0.8623s\n",
      "K Nearest Neighbors (KNN) ROC:0.406, precision @ rank n:0.0, execution time: 1.9502s\n",
      "Local Outlier Factor (LOF) ROC:0.5277, precision @ rank n:0.0303, execution time: 1.7256s\n",
      "Minimum Covariance Determinant (MCD) ROC:0.3822, precision @ rank n:0.0, execution time: 1.6714s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "One-class SVM (OCSVM) ROC:0.5171, precision @ rank n:0.0, execution time: 1.4729s\n",
      "Principal Component Analysis (PCA) ROC:0.526, precision @ rank n:0.0, execution time: 0.0521s\n",
      "\n",
      "... Processing pendigits.mat ...\n",
      "Angle-based Outlier Detector (ABOD) ROC:0.667, precision @ rank n:0.0526, execution time: 1.3767s\n",
      "Cluster-based Local Outlier Factor ROC:0.9172, precision @ rank n:0.1579, execution time: 0.1955s\n",
      "Feature Bagging ROC:0.4889, precision @ rank n:0.0526, execution time: 3.6236s\n",
      "Histogram-base Outlier Detection (HBOS) ROC:0.9348, precision @ rank n:0.2632, execution time: 0.008s\n",
      "Isolation Forest ROC:0.9376, precision @ rank n:0.3333, execution time: 0.6768s\n",
      "K Nearest Neighbors (KNN) ROC:0.7371, precision @ rank n:0.0702, execution time: 0.8352s\n",
      "Local Outlier Factor (LOF) ROC:0.4965, precision @ rank n:0.0702, execution time: 0.5886s\n",
      "Minimum Covariance Determinant (MCD) ROC:0.8204, precision @ rank n:0.0877, execution time: 2.274s\n",
      "One-class SVM (OCSVM) ROC:0.9235, precision @ rank n:0.3158, execution time: 0.9646s\n",
      "Principal Component Analysis (PCA) ROC:0.9309, precision @ rank n:0.3158, execution time: 0.007s\n",
      "\n",
      "... Processing pima.mat ...\n",
      "Angle-based Outlier Detector (ABOD) ROC:0.7163, precision @ rank n:0.5253, execution time: 0.1484s\n",
      "Cluster-based Local Outlier Factor ROC:0.7661, precision @ rank n:0.6061, execution time: 0.0752s\n",
      "Feature Bagging ROC:0.6448, precision @ rank n:0.4444, execution time: 0.0982s\n",
      "Histogram-base Outlier Detection (HBOS) ROC:0.711, precision @ rank n:0.5354, execution time: 0.002s\n",
      "Isolation Forest ROC:0.6818, precision @ rank n:0.5152, execution time: 0.2176s\n",
      "K Nearest Neighbors (KNN) ROC:0.7395, precision @ rank n:0.5859, execution time: 0.0572s\n",
      "Local Outlier Factor (LOF) ROC:0.6574, precision @ rank n:0.4646, execution time: 0.011s\n",
      "Minimum Covariance Determinant (MCD) ROC:0.7175, precision @ rank n:0.5152, execution time: 0.0491s\n",
      "One-class SVM (OCSVM) ROC:0.6561, precision @ rank n:0.5051, execution time: 0.01s\n",
      "Principal Component Analysis (PCA) ROC:0.6762, precision @ rank n:0.5354, execution time: 0.001s\n",
      "\n",
      "... Processing satellite.mat ...\n",
      "Angle-based Outlier Detector (ABOD) ROC:0.5653, precision @ rank n:0.3962, execution time: 1.6945s\n",
      "Cluster-based Local Outlier Factor ROC:0.5548, precision @ rank n:0.345, execution time: 0.3288s\n",
      "Feature Bagging ROC:0.572, precision @ rank n:0.4, execution time: 7.0868s\n",
      "Histogram-base Outlier Detection (HBOS) ROC:0.7486, precision @ rank n:0.57, execution time: 0.017s\n",
      "Isolation Forest ROC:0.6825, precision @ rank n:0.5825, execution time: 0.5966s\n",
      "K Nearest Neighbors (KNN) ROC:0.6853, precision @ rank n:0.4988, execution time: 1.1901s\n",
      "Local Outlier Factor (LOF) ROC:0.572, precision @ rank n:0.395, execution time: 0.9465s\n",
      "Minimum Covariance Determinant (MCD) ROC:0.8055, precision @ rank n:0.6762, execution time: 2.5738s\n",
      "One-class SVM (OCSVM) ROC:0.6478, precision @ rank n:0.5225, execution time: 1.3165s\n",
      "Principal Component Analysis (PCA) ROC:0.5923, precision @ rank n:0.465, execution time: 0.022s\n",
      "\n",
      "... Processing satimage-2.mat ...\n",
      "Angle-based Outlier Detector (ABOD) ROC:0.8432, precision @ rank n:0.2333, execution time: 1.3987s\n",
      "Cluster-based Local Outlier Factor ROC:0.9783, precision @ rank n:0.6667, execution time: 0.2396s\n",
      "Feature Bagging ROC:0.5235, precision @ rank n:0.1667, execution time: 5.8847s\n",
      "Histogram-base Outlier Detection (HBOS) ROC:0.9784, precision @ rank n:0.6, execution time: 0.015s\n",
      "Isolation Forest ROC:0.9952, precision @ rank n:0.8667, execution time: 0.5154s\n",
      "K Nearest Neighbors (KNN) ROC:0.9515, precision @ rank n:0.4333, execution time: 0.9897s\n",
      "Local Outlier Factor (LOF) ROC:0.5257, precision @ rank n:0.1667, execution time: 0.7229s\n",
      "Minimum Covariance Determinant (MCD) ROC:0.9963, precision @ rank n:0.6667, execution time: 2.0875s\n",
      "One-class SVM (OCSVM) ROC:0.9997, precision @ rank n:0.9, execution time: 1.0097s\n",
      "Principal Component Analysis (PCA) ROC:0.9816, precision @ rank n:0.7333, execution time: 0.016s\n",
      "\n",
      "... Processing shuttle.mat ...\n",
      "Angle-based Outlier Detector (ABOD) ROC:0.6171, precision @ rank n:0.2003, execution time: 13.0948s\n",
      "Cluster-based Local Outlier Factor ROC:0.6273, precision @ rank n:0.2025, execution time: 0.7941s\n",
      "Feature Bagging ROC:0.4725, precision @ rank n:0.0257, execution time: 73.3649s\n",
      "Histogram-base Outlier Detection (HBOS) ROC:0.9871, precision @ rank n:0.9985, execution time: 0.015s\n",
      "Isolation Forest ROC:0.9976, precision @ rank n:0.9596, execution time: 3.2025s\n",
      "K Nearest Neighbors (KNN) ROC:0.6507, precision @ rank n:0.212, execution time: 9.6808s\n",
      "Local Outlier Factor (LOF) ROC:0.5556, precision @ rank n:0.1548, execution time: 10.6754s\n",
      "Minimum Covariance Determinant (MCD) ROC:0.99, precision @ rank n:0.7395, execution time: 10.3525s\n",
      "One-class SVM (OCSVM) ROC:0.9934, precision @ rank n:0.956, execution time: 44.1575s\n",
      "Principal Component Analysis (PCA) ROC:0.9915, precision @ rank n:0.9516, execution time: 0.032s\n",
      "\n",
      "... Processing vertebral.mat ...\n",
      "Angle-based Outlier Detector (ABOD) ROC:0.5366, precision @ rank n:0.2143, execution time: 0.0431s\n",
      "Cluster-based Local Outlier Factor ROC:0.3937, precision @ rank n:0.0, execution time: 0.0311s\n",
      "Feature Bagging ROC:0.5279, precision @ rank n:0.1429, execution time: 0.029s\n",
      "Histogram-base Outlier Detection (HBOS) ROC:0.3506, precision @ rank n:0.0, execution time: 0.002s\n",
      "Isolation Forest ROC:0.3772, precision @ rank n:0.0, execution time: 0.1575s\n",
      "K Nearest Neighbors (KNN) ROC:0.4573, precision @ rank n:0.0714, execution time: 0.015s\n",
      "Local Outlier Factor (LOF) ROC:0.4983, precision @ rank n:0.1429, execution time: 0.003s\n",
      "Minimum Covariance Determinant (MCD) ROC:0.4103, precision @ rank n:0.0714, execution time: 0.0331s\n",
      "One-class SVM (OCSVM) ROC:0.4686, precision @ rank n:0.0714, execution time: 0.002s\n",
      "Principal Component Analysis (PCA) ROC:0.4085, precision @ rank n:0.0, execution time: 0.001s\n",
      "\n",
      "... Processing vowels.mat ...\n",
      "Angle-based Outlier Detector (ABOD) ROC:0.9616, precision @ rank n:0.6316, execution time: 0.2336s\n",
      "Cluster-based Local Outlier Factor ROC:0.6496, precision @ rank n:0.1053, execution time: 0.0662s\n",
      "Feature Bagging ROC:0.9365, precision @ rank n:0.3684, execution time: 0.2577s\n",
      "Histogram-base Outlier Detection (HBOS) ROC:0.6876, precision @ rank n:0.1579, execution time: 0.003s\n",
      "Isolation Forest ROC:0.8209, precision @ rank n:0.1579, execution time: 0.2116s\n",
      "K Nearest Neighbors (KNN) ROC:0.9734, precision @ rank n:0.4737, execution time: 0.1113s\n",
      "Local Outlier Factor (LOF) ROC:0.9398, precision @ rank n:0.3684, execution time: 0.0301s\n",
      "Minimum Covariance Determinant (MCD) ROC:0.7243, precision @ rank n:0.1053, execution time: 0.755s\n",
      "One-class SVM (OCSVM) ROC:0.8163, precision @ rank n:0.2632, execution time: 0.0311s\n",
      "Principal Component Analysis (PCA) ROC:0.6297, precision @ rank n:0.1579, execution time: 0.002s\n",
      "\n",
      "... Processing wbc.mat ...\n",
      "Angle-based Outlier Detector (ABOD) ROC:0.921, precision @ rank n:0.375, execution time: 0.0632s\n",
      "Cluster-based Local Outlier Factor ROC:0.8906, precision @ rank n:0.375, execution time: 0.0421s\n",
      "Feature Bagging ROC:0.9271, precision @ rank n:0.375, execution time: 0.0792s\n",
      "Histogram-base Outlier Detection (HBOS) ROC:0.9479, precision @ rank n:0.5, execution time: 0.005s\n",
      "Isolation Forest ROC:0.9436, precision @ rank n:0.5, execution time: 0.1564s\n",
      "K Nearest Neighbors (KNN) ROC:0.9444, precision @ rank n:0.5, execution time: 0.0271s\n",
      "Local Outlier Factor (LOF) ROC:0.9227, precision @ rank n:0.375, execution time: 0.007s\n",
      "Minimum Covariance Determinant (MCD) ROC:0.9288, precision @ rank n:0.5, execution time: 0.0632s\n",
      "One-class SVM (OCSVM) ROC:0.9358, precision @ rank n:0.375, execution time: 0.005s\n",
      "Principal Component Analysis (PCA) ROC:0.9262, precision @ rank n:0.375, execution time: 0.001s\n"
     ]
    }
   ],
   "source": [
    "# Define data file and read X and y\n",
    "mat_file_list = ['arrhythmia.mat',\n",
    "                 'cardio.mat',\n",
    "                 'glass.mat',\n",
    "                 'ionosphere.mat',\n",
    "                 'letter.mat',\n",
    "                 'lympho.mat',\n",
    "                 'mnist.mat',\n",
    "                 'musk.mat',\n",
    "                 'optdigits.mat',\n",
    "                 'pendigits.mat',\n",
    "                 'pima.mat',\n",
    "                 'satellite.mat',\n",
    "                 'satimage-2.mat',\n",
    "                 'shuttle.mat',\n",
    "                 'vertebral.mat',\n",
    "                 'vowels.mat',\n",
    "                 'wbc.mat']\n",
    "\n",
    "# Define nine outlier detection tools to be compared\n",
    "random_state = np.random.RandomState(42)\n",
    "\n",
    "df_columns = ['Data', '#Samples', '# Dimensions', 'Outlier Perc',\n",
    "              'ABOD', 'CBLOF', 'FB', 'HBOS', 'IForest', 'KNN', 'LOF', 'MCD',\n",
    "              'OCSVM', 'PCA']\n",
    "roc_df = pd.DataFrame(columns=df_columns)\n",
    "prn_df = pd.DataFrame(columns=df_columns)\n",
    "time_df = pd.DataFrame(columns=df_columns)\n",
    "\n",
    "\n",
    "for mat_file in mat_file_list:\n",
    "    print(\"\\n... Processing\", mat_file, '...')\n",
    "    mat = loadmat(os.path.join('data', mat_file))\n",
    "\n",
    "    X = mat['X']\n",
    "    y = mat['y'].ravel()\n",
    "    outliers_fraction = np.count_nonzero(y) / len(y)\n",
    "    outliers_percentage = round(outliers_fraction * 100, ndigits=4)\n",
    "\n",
    "    # construct containers for saving results\n",
    "    roc_list = [mat_file[:-4], X.shape[0], X.shape[1], outliers_percentage]\n",
    "    prn_list = [mat_file[:-4], X.shape[0], X.shape[1], outliers_percentage]\n",
    "    time_list = [mat_file[:-4], X.shape[0], X.shape[1], outliers_percentage]\n",
    "\n",
    "    # 60% data for training and 40% for testing\n",
    "    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4,\n",
    "                                                        random_state=random_state)\n",
    "\n",
    "    # standardizing data for processing\n",
    "    X_train_norm, X_test_norm = standardizer(X_train, X_test)\n",
    "\n",
    "    classifiers = {'Angle-based Outlier Detector (ABOD)': ABOD(\n",
    "        contamination=outliers_fraction),\n",
    "        'Cluster-based Local Outlier Factor': CBLOF(\n",
    "            contamination=outliers_fraction, check_estimator=False,\n",
    "            random_state=random_state),\n",
    "        'Feature Bagging': FeatureBagging(contamination=outliers_fraction,\n",
    "                                          random_state=random_state),\n",
    "        'Histogram-base Outlier Detection (HBOS)': HBOS(\n",
    "            contamination=outliers_fraction),\n",
    "        'Isolation Forest': IForest(contamination=outliers_fraction,\n",
    "                                    random_state=random_state),\n",
    "        'K Nearest Neighbors (KNN)': KNN(contamination=outliers_fraction),\n",
    "        'Local Outlier Factor (LOF)': LOF(\n",
    "            contamination=outliers_fraction),\n",
    "        'Minimum Covariance Determinant (MCD)': MCD(\n",
    "            contamination=outliers_fraction, random_state=random_state),\n",
    "        'One-class SVM (OCSVM)': OCSVM(contamination=outliers_fraction),\n",
    "        'Principal Component Analysis (PCA)': PCA(\n",
    "            contamination=outliers_fraction, random_state=random_state),\n",
    "    }\n",
    "\n",
    "    for clf_name, clf in classifiers.items():\n",
    "        t0 = time()\n",
    "        clf.fit(X_train_norm)\n",
    "        test_scores = clf.decision_function(X_test_norm)\n",
    "        t1 = time()\n",
    "        duration = round(t1 - t0, ndigits=4)\n",
    "        time_list.append(duration)\n",
    "\n",
    "        roc = round(roc_auc_score(y_test, test_scores), ndigits=4)\n",
    "        prn = round(precision_n_scores(y_test, test_scores), ndigits=4)\n",
    "\n",
    "        print('{clf_name} ROC:{roc}, precision @ rank n:{prn}, '\n",
    "              'execution time: {duration}s'.format(\n",
    "            clf_name=clf_name, roc=roc, prn=prn, duration=duration))\n",
    "\n",
    "        roc_list.append(roc)\n",
    "        prn_list.append(prn)\n",
    "\n",
    "    temp_df = pd.DataFrame(time_list).transpose()\n",
    "    temp_df.columns = df_columns\n",
    "    time_df = pd.concat([time_df, temp_df], axis=0)\n",
    "\n",
    "    temp_df = pd.DataFrame(roc_list).transpose()\n",
    "    temp_df.columns = df_columns\n",
    "    roc_df = pd.concat([roc_df, temp_df], axis=0)\n",
    "\n",
    "    temp_df = pd.DataFrame(prn_list).transpose()\n",
    "    temp_df.columns = df_columns\n",
    "    prn_df = pd.concat([prn_df, temp_df], axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time complexity\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Data</th>\n",
       "      <th>#Samples</th>\n",
       "      <th># Dimensions</th>\n",
       "      <th>Outlier Perc</th>\n",
       "      <th>ABOD</th>\n",
       "      <th>CBLOF</th>\n",
       "      <th>FB</th>\n",
       "      <th>HBOS</th>\n",
       "      <th>IForest</th>\n",
       "      <th>KNN</th>\n",
       "      <th>LOF</th>\n",
       "      <th>MCD</th>\n",
       "      <th>OCSVM</th>\n",
       "      <th>PCA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>arrhythmia</td>\n",
       "      <td>452</td>\n",
       "      <td>274</td>\n",
       "      <td>14.6018</td>\n",
       "      <td>1.126</td>\n",
       "      <td>0.9786</td>\n",
       "      <td>0.6076</td>\n",
       "      <td>0.8914</td>\n",
       "      <td>0.2507</td>\n",
       "      <td>0.1133</td>\n",
       "      <td>0.0742</td>\n",
       "      <td>1.4578</td>\n",
       "      <td>0.0481</td>\n",
       "      <td>0.0642</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>cardio</td>\n",
       "      <td>1831</td>\n",
       "      <td>21</td>\n",
       "      <td>9.6122</td>\n",
       "      <td>0.3579</td>\n",
       "      <td>0.0983</td>\n",
       "      <td>0.8683</td>\n",
       "      <td>0.005</td>\n",
       "      <td>0.2767</td>\n",
       "      <td>0.1905</td>\n",
       "      <td>0.0993</td>\n",
       "      <td>0.5835</td>\n",
       "      <td>0.0822</td>\n",
       "      <td>0.003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>glass</td>\n",
       "      <td>214</td>\n",
       "      <td>9</td>\n",
       "      <td>4.2056</td>\n",
       "      <td>0.0361</td>\n",
       "      <td>0.0251</td>\n",
       "      <td>0.0351</td>\n",
       "      <td>0.005</td>\n",
       "      <td>0.1584</td>\n",
       "      <td>0.013</td>\n",
       "      <td>0.003</td>\n",
       "      <td>0.0481</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ionosphere</td>\n",
       "      <td>351</td>\n",
       "      <td>33</td>\n",
       "      <td>35.8974</td>\n",
       "      <td>0.0602</td>\n",
       "      <td>0.0381</td>\n",
       "      <td>0.0642</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.1785</td>\n",
       "      <td>0.0251</td>\n",
       "      <td>0.007</td>\n",
       "      <td>0.0622</td>\n",
       "      <td>0.004</td>\n",
       "      <td>0.003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>letter</td>\n",
       "      <td>1600</td>\n",
       "      <td>32</td>\n",
       "      <td>6.25</td>\n",
       "      <td>0.365</td>\n",
       "      <td>0.1052</td>\n",
       "      <td>0.7309</td>\n",
       "      <td>0.007</td>\n",
       "      <td>0.2416</td>\n",
       "      <td>0.1905</td>\n",
       "      <td>0.0922</td>\n",
       "      <td>1.3225</td>\n",
       "      <td>0.0973</td>\n",
       "      <td>0.005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>lympho</td>\n",
       "      <td>148</td>\n",
       "      <td>18</td>\n",
       "      <td>4.0541</td>\n",
       "      <td>0.0261</td>\n",
       "      <td>0.0321</td>\n",
       "      <td>0.0261</td>\n",
       "      <td>0.005</td>\n",
       "      <td>0.1795</td>\n",
       "      <td>0.011</td>\n",
       "      <td>0.003</td>\n",
       "      <td>0.0531</td>\n",
       "      <td>0.002</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mnist</td>\n",
       "      <td>7603</td>\n",
       "      <td>100</td>\n",
       "      <td>9.2069</td>\n",
       "      <td>6.8502</td>\n",
       "      <td>0.9004</td>\n",
       "      <td>44.9604</td>\n",
       "      <td>0.0411</td>\n",
       "      <td>1.6943</td>\n",
       "      <td>6.6327</td>\n",
       "      <td>5.9395</td>\n",
       "      <td>4.1711</td>\n",
       "      <td>4.3626</td>\n",
       "      <td>0.1433</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>musk</td>\n",
       "      <td>3062</td>\n",
       "      <td>166</td>\n",
       "      <td>3.1679</td>\n",
       "      <td>1.8951</td>\n",
       "      <td>0.2998</td>\n",
       "      <td>12.2746</td>\n",
       "      <td>0.0582</td>\n",
       "      <td>1.0528</td>\n",
       "      <td>1.7376</td>\n",
       "      <td>1.5411</td>\n",
       "      <td>21.114</td>\n",
       "      <td>1.1922</td>\n",
       "      <td>0.1504</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>optdigits</td>\n",
       "      <td>5216</td>\n",
       "      <td>64</td>\n",
       "      <td>2.8758</td>\n",
       "      <td>2.243</td>\n",
       "      <td>0.3911</td>\n",
       "      <td>11.9945</td>\n",
       "      <td>0.0271</td>\n",
       "      <td>0.8623</td>\n",
       "      <td>1.9502</td>\n",
       "      <td>1.7256</td>\n",
       "      <td>1.6714</td>\n",
       "      <td>1.4729</td>\n",
       "      <td>0.0521</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>pendigits</td>\n",
       "      <td>6870</td>\n",
       "      <td>16</td>\n",
       "      <td>2.2707</td>\n",
       "      <td>1.3767</td>\n",
       "      <td>0.1955</td>\n",
       "      <td>3.6236</td>\n",
       "      <td>0.008</td>\n",
       "      <td>0.6768</td>\n",
       "      <td>0.8352</td>\n",
       "      <td>0.5886</td>\n",
       "      <td>2.274</td>\n",
       "      <td>0.9646</td>\n",
       "      <td>0.007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>pima</td>\n",
       "      <td>768</td>\n",
       "      <td>8</td>\n",
       "      <td>34.8958</td>\n",
       "      <td>0.1484</td>\n",
       "      <td>0.0752</td>\n",
       "      <td>0.0982</td>\n",
       "      <td>0.002</td>\n",
       "      <td>0.2176</td>\n",
       "      <td>0.0572</td>\n",
       "      <td>0.011</td>\n",
       "      <td>0.0491</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>satellite</td>\n",
       "      <td>6435</td>\n",
       "      <td>36</td>\n",
       "      <td>31.6395</td>\n",
       "      <td>1.6945</td>\n",
       "      <td>0.3288</td>\n",
       "      <td>7.0868</td>\n",
       "      <td>0.017</td>\n",
       "      <td>0.5966</td>\n",
       "      <td>1.1901</td>\n",
       "      <td>0.9465</td>\n",
       "      <td>2.5738</td>\n",
       "      <td>1.3165</td>\n",
       "      <td>0.022</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>satimage-2</td>\n",
       "      <td>5803</td>\n",
       "      <td>36</td>\n",
       "      <td>1.2235</td>\n",
       "      <td>1.3987</td>\n",
       "      <td>0.2396</td>\n",
       "      <td>5.8847</td>\n",
       "      <td>0.015</td>\n",
       "      <td>0.5154</td>\n",
       "      <td>0.9897</td>\n",
       "      <td>0.7229</td>\n",
       "      <td>2.0875</td>\n",
       "      <td>1.0097</td>\n",
       "      <td>0.016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>shuttle</td>\n",
       "      <td>49097</td>\n",
       "      <td>9</td>\n",
       "      <td>7.1511</td>\n",
       "      <td>13.0948</td>\n",
       "      <td>0.7941</td>\n",
       "      <td>73.3649</td>\n",
       "      <td>0.015</td>\n",
       "      <td>3.2025</td>\n",
       "      <td>9.6808</td>\n",
       "      <td>10.6754</td>\n",
       "      <td>10.3525</td>\n",
       "      <td>44.1575</td>\n",
       "      <td>0.032</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>vertebral</td>\n",
       "      <td>240</td>\n",
       "      <td>6</td>\n",
       "      <td>12.5</td>\n",
       "      <td>0.0431</td>\n",
       "      <td>0.0311</td>\n",
       "      <td>0.029</td>\n",
       "      <td>0.002</td>\n",
       "      <td>0.1575</td>\n",
       "      <td>0.015</td>\n",
       "      <td>0.003</td>\n",
       "      <td>0.0331</td>\n",
       "      <td>0.002</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>vowels</td>\n",
       "      <td>1456</td>\n",
       "      <td>12</td>\n",
       "      <td>3.4341</td>\n",
       "      <td>0.2336</td>\n",
       "      <td>0.0662</td>\n",
       "      <td>0.2577</td>\n",
       "      <td>0.003</td>\n",
       "      <td>0.2116</td>\n",
       "      <td>0.1113</td>\n",
       "      <td>0.0301</td>\n",
       "      <td>0.755</td>\n",
       "      <td>0.0311</td>\n",
       "      <td>0.002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>wbc</td>\n",
       "      <td>378</td>\n",
       "      <td>30</td>\n",
       "      <td>5.5556</td>\n",
       "      <td>0.0632</td>\n",
       "      <td>0.0421</td>\n",
       "      <td>0.0792</td>\n",
       "      <td>0.005</td>\n",
       "      <td>0.1564</td>\n",
       "      <td>0.0271</td>\n",
       "      <td>0.007</td>\n",
       "      <td>0.0632</td>\n",
       "      <td>0.005</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Data #Samples # Dimensions Outlier Perc     ABOD   CBLOF       FB  \\\n",
       "0  arrhythmia      452          274      14.6018    1.126  0.9786   0.6076   \n",
       "0      cardio     1831           21       9.6122   0.3579  0.0983   0.8683   \n",
       "0       glass      214            9       4.2056   0.0361  0.0251   0.0351   \n",
       "0  ionosphere      351           33      35.8974   0.0602  0.0381   0.0642   \n",
       "0      letter     1600           32         6.25    0.365  0.1052   0.7309   \n",
       "0      lympho      148           18       4.0541   0.0261  0.0321   0.0261   \n",
       "0       mnist     7603          100       9.2069   6.8502  0.9004  44.9604   \n",
       "0        musk     3062          166       3.1679   1.8951  0.2998  12.2746   \n",
       "0   optdigits     5216           64       2.8758    2.243  0.3911  11.9945   \n",
       "0   pendigits     6870           16       2.2707   1.3767  0.1955   3.6236   \n",
       "0        pima      768            8      34.8958   0.1484  0.0752   0.0982   \n",
       "0   satellite     6435           36      31.6395   1.6945  0.3288   7.0868   \n",
       "0  satimage-2     5803           36       1.2235   1.3987  0.2396   5.8847   \n",
       "0     shuttle    49097            9       7.1511  13.0948  0.7941  73.3649   \n",
       "0   vertebral      240            6         12.5   0.0431  0.0311    0.029   \n",
       "0      vowels     1456           12       3.4341   0.2336  0.0662   0.2577   \n",
       "0         wbc      378           30       5.5556   0.0632  0.0421   0.0792   \n",
       "\n",
       "     HBOS IForest     KNN      LOF      MCD    OCSVM     PCA  \n",
       "0  0.8914  0.2507  0.1133   0.0742   1.4578   0.0481  0.0642  \n",
       "0   0.005  0.2767  0.1905   0.0993   0.5835   0.0822   0.003  \n",
       "0   0.005  0.1584   0.013    0.003   0.0481    0.001   0.001  \n",
       "0   0.006  0.1785  0.0251    0.007   0.0622    0.004   0.003  \n",
       "0   0.007  0.2416  0.1905   0.0922   1.3225   0.0973   0.005  \n",
       "0   0.005  0.1795   0.011    0.003   0.0531    0.002   0.001  \n",
       "0  0.0411  1.6943  6.6327   5.9395   4.1711   4.3626  0.1433  \n",
       "0  0.0582  1.0528  1.7376   1.5411   21.114   1.1922  0.1504  \n",
       "0  0.0271  0.8623  1.9502   1.7256   1.6714   1.4729  0.0521  \n",
       "0   0.008  0.6768  0.8352   0.5886    2.274   0.9646   0.007  \n",
       "0   0.002  0.2176  0.0572    0.011   0.0491     0.01   0.001  \n",
       "0   0.017  0.5966  1.1901   0.9465   2.5738   1.3165   0.022  \n",
       "0   0.015  0.5154  0.9897   0.7229   2.0875   1.0097   0.016  \n",
       "0   0.015  3.2025  9.6808  10.6754  10.3525  44.1575   0.032  \n",
       "0   0.002  0.1575   0.015    0.003   0.0331    0.002   0.001  \n",
       "0   0.003  0.2116  0.1113   0.0301    0.755   0.0311   0.002  \n",
       "0   0.005  0.1564  0.0271    0.007   0.0632    0.005   0.001  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Time complexity')\n",
    "time_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Analyze the performance of ROC and Precision @ n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ROC Performance\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Data</th>\n",
       "      <th>#Samples</th>\n",
       "      <th># Dimensions</th>\n",
       "      <th>Outlier Perc</th>\n",
       "      <th>ABOD</th>\n",
       "      <th>CBLOF</th>\n",
       "      <th>FB</th>\n",
       "      <th>HBOS</th>\n",
       "      <th>IForest</th>\n",
       "      <th>KNN</th>\n",
       "      <th>LOF</th>\n",
       "      <th>MCD</th>\n",
       "      <th>OCSVM</th>\n",
       "      <th>PCA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>arrhythmia</td>\n",
       "      <td>452</td>\n",
       "      <td>274</td>\n",
       "      <td>14.6018</td>\n",
       "      <td>0.7687</td>\n",
       "      <td>0.7824</td>\n",
       "      <td>0.7796</td>\n",
       "      <td>0.8511</td>\n",
       "      <td>0.8639</td>\n",
       "      <td>0.782</td>\n",
       "      <td>0.7787</td>\n",
       "      <td>0.8228</td>\n",
       "      <td>0.7986</td>\n",
       "      <td>0.7997</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>cardio</td>\n",
       "      <td>1831</td>\n",
       "      <td>21</td>\n",
       "      <td>9.6122</td>\n",
       "      <td>0.5892</td>\n",
       "      <td>0.973</td>\n",
       "      <td>0.6385</td>\n",
       "      <td>0.8373</td>\n",
       "      <td>0.9502</td>\n",
       "      <td>0.734</td>\n",
       "      <td>0.588</td>\n",
       "      <td>0.8195</td>\n",
       "      <td>0.9478</td>\n",
       "      <td>0.9616</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>glass</td>\n",
       "      <td>214</td>\n",
       "      <td>9</td>\n",
       "      <td>4.2056</td>\n",
       "      <td>0.6951</td>\n",
       "      <td>0.7957</td>\n",
       "      <td>0.7073</td>\n",
       "      <td>0.7073</td>\n",
       "      <td>0.7134</td>\n",
       "      <td>0.8384</td>\n",
       "      <td>0.7043</td>\n",
       "      <td>0.8293</td>\n",
       "      <td>0.6585</td>\n",
       "      <td>0.686</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ionosphere</td>\n",
       "      <td>351</td>\n",
       "      <td>33</td>\n",
       "      <td>35.8974</td>\n",
       "      <td>0.9181</td>\n",
       "      <td>0.795</td>\n",
       "      <td>0.9303</td>\n",
       "      <td>0.6052</td>\n",
       "      <td>0.8486</td>\n",
       "      <td>0.932</td>\n",
       "      <td>0.9227</td>\n",
       "      <td>0.9669</td>\n",
       "      <td>0.8257</td>\n",
       "      <td>0.7941</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>letter</td>\n",
       "      <td>1600</td>\n",
       "      <td>32</td>\n",
       "      <td>6.25</td>\n",
       "      <td>0.8783</td>\n",
       "      <td>0.5301</td>\n",
       "      <td>0.8947</td>\n",
       "      <td>0.6063</td>\n",
       "      <td>0.6201</td>\n",
       "      <td>0.8573</td>\n",
       "      <td>0.8765</td>\n",
       "      <td>0.8061</td>\n",
       "      <td>0.5927</td>\n",
       "      <td>0.5216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>lympho</td>\n",
       "      <td>148</td>\n",
       "      <td>18</td>\n",
       "      <td>4.0541</td>\n",
       "      <td>0.9831</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mnist</td>\n",
       "      <td>7603</td>\n",
       "      <td>100</td>\n",
       "      <td>9.2069</td>\n",
       "      <td>0.7628</td>\n",
       "      <td>0.8204</td>\n",
       "      <td>0.7157</td>\n",
       "      <td>0.5766</td>\n",
       "      <td>0.7939</td>\n",
       "      <td>0.8498</td>\n",
       "      <td>0.7195</td>\n",
       "      <td>0.8713</td>\n",
       "      <td>0.854</td>\n",
       "      <td>0.8534</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>musk</td>\n",
       "      <td>3062</td>\n",
       "      <td>166</td>\n",
       "      <td>3.1679</td>\n",
       "      <td>0.2161</td>\n",
       "      <td>0.9899</td>\n",
       "      <td>0.473</td>\n",
       "      <td>0.9999</td>\n",
       "      <td>1</td>\n",
       "      <td>0.8009</td>\n",
       "      <td>0.4629</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>optdigits</td>\n",
       "      <td>5216</td>\n",
       "      <td>64</td>\n",
       "      <td>2.8758</td>\n",
       "      <td>0.4894</td>\n",
       "      <td>0.5329</td>\n",
       "      <td>0.5062</td>\n",
       "      <td>0.8774</td>\n",
       "      <td>0.6735</td>\n",
       "      <td>0.406</td>\n",
       "      <td>0.5277</td>\n",
       "      <td>0.3822</td>\n",
       "      <td>0.5171</td>\n",
       "      <td>0.526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>pendigits</td>\n",
       "      <td>6870</td>\n",
       "      <td>16</td>\n",
       "      <td>2.2707</td>\n",
       "      <td>0.667</td>\n",
       "      <td>0.9172</td>\n",
       "      <td>0.4889</td>\n",
       "      <td>0.9348</td>\n",
       "      <td>0.9376</td>\n",
       "      <td>0.7371</td>\n",
       "      <td>0.4965</td>\n",
       "      <td>0.8204</td>\n",
       "      <td>0.9235</td>\n",
       "      <td>0.9309</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>pima</td>\n",
       "      <td>768</td>\n",
       "      <td>8</td>\n",
       "      <td>34.8958</td>\n",
       "      <td>0.7163</td>\n",
       "      <td>0.7661</td>\n",
       "      <td>0.6448</td>\n",
       "      <td>0.711</td>\n",
       "      <td>0.6818</td>\n",
       "      <td>0.7395</td>\n",
       "      <td>0.6574</td>\n",
       "      <td>0.7175</td>\n",
       "      <td>0.6561</td>\n",
       "      <td>0.6762</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>satellite</td>\n",
       "      <td>6435</td>\n",
       "      <td>36</td>\n",
       "      <td>31.6395</td>\n",
       "      <td>0.5653</td>\n",
       "      <td>0.5548</td>\n",
       "      <td>0.572</td>\n",
       "      <td>0.7486</td>\n",
       "      <td>0.6825</td>\n",
       "      <td>0.6853</td>\n",
       "      <td>0.572</td>\n",
       "      <td>0.8055</td>\n",
       "      <td>0.6478</td>\n",
       "      <td>0.5923</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>satimage-2</td>\n",
       "      <td>5803</td>\n",
       "      <td>36</td>\n",
       "      <td>1.2235</td>\n",
       "      <td>0.8432</td>\n",
       "      <td>0.9783</td>\n",
       "      <td>0.5235</td>\n",
       "      <td>0.9784</td>\n",
       "      <td>0.9952</td>\n",
       "      <td>0.9515</td>\n",
       "      <td>0.5257</td>\n",
       "      <td>0.9963</td>\n",
       "      <td>0.9997</td>\n",
       "      <td>0.9816</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>shuttle</td>\n",
       "      <td>49097</td>\n",
       "      <td>9</td>\n",
       "      <td>7.1511</td>\n",
       "      <td>0.6171</td>\n",
       "      <td>0.6273</td>\n",
       "      <td>0.4725</td>\n",
       "      <td>0.9871</td>\n",
       "      <td>0.9976</td>\n",
       "      <td>0.6507</td>\n",
       "      <td>0.5556</td>\n",
       "      <td>0.99</td>\n",
       "      <td>0.9934</td>\n",
       "      <td>0.9915</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>vertebral</td>\n",
       "      <td>240</td>\n",
       "      <td>6</td>\n",
       "      <td>12.5</td>\n",
       "      <td>0.5366</td>\n",
       "      <td>0.3937</td>\n",
       "      <td>0.5279</td>\n",
       "      <td>0.3506</td>\n",
       "      <td>0.3772</td>\n",
       "      <td>0.4573</td>\n",
       "      <td>0.4983</td>\n",
       "      <td>0.4103</td>\n",
       "      <td>0.4686</td>\n",
       "      <td>0.4085</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>vowels</td>\n",
       "      <td>1456</td>\n",
       "      <td>12</td>\n",
       "      <td>3.4341</td>\n",
       "      <td>0.9616</td>\n",
       "      <td>0.6496</td>\n",
       "      <td>0.9365</td>\n",
       "      <td>0.6876</td>\n",
       "      <td>0.8209</td>\n",
       "      <td>0.9734</td>\n",
       "      <td>0.9398</td>\n",
       "      <td>0.7243</td>\n",
       "      <td>0.8163</td>\n",
       "      <td>0.6297</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>wbc</td>\n",
       "      <td>378</td>\n",
       "      <td>30</td>\n",
       "      <td>5.5556</td>\n",
       "      <td>0.921</td>\n",
       "      <td>0.8906</td>\n",
       "      <td>0.9271</td>\n",
       "      <td>0.9479</td>\n",
       "      <td>0.9436</td>\n",
       "      <td>0.9444</td>\n",
       "      <td>0.9227</td>\n",
       "      <td>0.9288</td>\n",
       "      <td>0.9358</td>\n",
       "      <td>0.9262</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Data #Samples # Dimensions Outlier Perc    ABOD   CBLOF      FB  \\\n",
       "0  arrhythmia      452          274      14.6018  0.7687  0.7824  0.7796   \n",
       "0      cardio     1831           21       9.6122  0.5892   0.973  0.6385   \n",
       "0       glass      214            9       4.2056  0.6951  0.7957  0.7073   \n",
       "0  ionosphere      351           33      35.8974  0.9181   0.795  0.9303   \n",
       "0      letter     1600           32         6.25  0.8783  0.5301  0.8947   \n",
       "0      lympho      148           18       4.0541  0.9831       1       1   \n",
       "0       mnist     7603          100       9.2069  0.7628  0.8204  0.7157   \n",
       "0        musk     3062          166       3.1679  0.2161  0.9899   0.473   \n",
       "0   optdigits     5216           64       2.8758  0.4894  0.5329  0.5062   \n",
       "0   pendigits     6870           16       2.2707   0.667  0.9172  0.4889   \n",
       "0        pima      768            8      34.8958  0.7163  0.7661  0.6448   \n",
       "0   satellite     6435           36      31.6395  0.5653  0.5548   0.572   \n",
       "0  satimage-2     5803           36       1.2235  0.8432  0.9783  0.5235   \n",
       "0     shuttle    49097            9       7.1511  0.6171  0.6273  0.4725   \n",
       "0   vertebral      240            6         12.5  0.5366  0.3937  0.5279   \n",
       "0      vowels     1456           12       3.4341  0.9616  0.6496  0.9365   \n",
       "0         wbc      378           30       5.5556   0.921  0.8906  0.9271   \n",
       "\n",
       "     HBOS IForest     KNN     LOF     MCD   OCSVM     PCA  \n",
       "0  0.8511  0.8639   0.782  0.7787  0.8228  0.7986  0.7997  \n",
       "0  0.8373  0.9502   0.734   0.588  0.8195  0.9478  0.9616  \n",
       "0  0.7073  0.7134  0.8384  0.7043  0.8293  0.6585   0.686  \n",
       "0  0.6052  0.8486   0.932  0.9227  0.9669  0.8257  0.7941  \n",
       "0  0.6063  0.6201  0.8573  0.8765  0.8061  0.5927  0.5216  \n",
       "0       1       1       1       1       1       1       1  \n",
       "0  0.5766  0.7939  0.8498  0.7195  0.8713   0.854  0.8534  \n",
       "0  0.9999       1  0.8009  0.4629       1       1       1  \n",
       "0  0.8774  0.6735   0.406  0.5277  0.3822  0.5171   0.526  \n",
       "0  0.9348  0.9376  0.7371  0.4965  0.8204  0.9235  0.9309  \n",
       "0   0.711  0.6818  0.7395  0.6574  0.7175  0.6561  0.6762  \n",
       "0  0.7486  0.6825  0.6853   0.572  0.8055  0.6478  0.5923  \n",
       "0  0.9784  0.9952  0.9515  0.5257  0.9963  0.9997  0.9816  \n",
       "0  0.9871  0.9976  0.6507  0.5556    0.99  0.9934  0.9915  \n",
       "0  0.3506  0.3772  0.4573  0.4983  0.4103  0.4686  0.4085  \n",
       "0  0.6876  0.8209  0.9734  0.9398  0.7243  0.8163  0.6297  \n",
       "0  0.9479  0.9436  0.9444  0.9227  0.9288  0.9358  0.9262  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('ROC Performance')\n",
    "roc_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision @ n Performance\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Data</th>\n",
       "      <th>#Samples</th>\n",
       "      <th># Dimensions</th>\n",
       "      <th>Outlier Perc</th>\n",
       "      <th>ABOD</th>\n",
       "      <th>CBLOF</th>\n",
       "      <th>FB</th>\n",
       "      <th>HBOS</th>\n",
       "      <th>IForest</th>\n",
       "      <th>KNN</th>\n",
       "      <th>LOF</th>\n",
       "      <th>MCD</th>\n",
       "      <th>OCSVM</th>\n",
       "      <th>PCA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>arrhythmia</td>\n",
       "      <td>452</td>\n",
       "      <td>274</td>\n",
       "      <td>14.6018</td>\n",
       "      <td>0.3571</td>\n",
       "      <td>0.4643</td>\n",
       "      <td>0.4643</td>\n",
       "      <td>0.5714</td>\n",
       "      <td>0.6071</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.4643</td>\n",
       "      <td>0.4286</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>cardio</td>\n",
       "      <td>1831</td>\n",
       "      <td>21</td>\n",
       "      <td>9.6122</td>\n",
       "      <td>0.1918</td>\n",
       "      <td>0.7945</td>\n",
       "      <td>0.1781</td>\n",
       "      <td>0.4521</td>\n",
       "      <td>0.6027</td>\n",
       "      <td>0.3562</td>\n",
       "      <td>0.1507</td>\n",
       "      <td>0.411</td>\n",
       "      <td>0.5342</td>\n",
       "      <td>0.6849</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>glass</td>\n",
       "      <td>214</td>\n",
       "      <td>9</td>\n",
       "      <td>4.2056</td>\n",
       "      <td>0.25</td>\n",
       "      <td>0.25</td>\n",
       "      <td>0.25</td>\n",
       "      <td>0</td>\n",
       "      <td>0.25</td>\n",
       "      <td>0.25</td>\n",
       "      <td>0.25</td>\n",
       "      <td>0</td>\n",
       "      <td>0.25</td>\n",
       "      <td>0.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ionosphere</td>\n",
       "      <td>351</td>\n",
       "      <td>33</td>\n",
       "      <td>35.8974</td>\n",
       "      <td>0.8431</td>\n",
       "      <td>0.549</td>\n",
       "      <td>0.8039</td>\n",
       "      <td>0.3922</td>\n",
       "      <td>0.5882</td>\n",
       "      <td>0.8824</td>\n",
       "      <td>0.7843</td>\n",
       "      <td>0.8627</td>\n",
       "      <td>0.6863</td>\n",
       "      <td>0.5686</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>letter</td>\n",
       "      <td>1600</td>\n",
       "      <td>32</td>\n",
       "      <td>6.25</td>\n",
       "      <td>0.4375</td>\n",
       "      <td>0.0312</td>\n",
       "      <td>0.4062</td>\n",
       "      <td>0.0938</td>\n",
       "      <td>0.0625</td>\n",
       "      <td>0.3125</td>\n",
       "      <td>0.3438</td>\n",
       "      <td>0.1875</td>\n",
       "      <td>0.125</td>\n",
       "      <td>0.125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>lympho</td>\n",
       "      <td>148</td>\n",
       "      <td>18</td>\n",
       "      <td>4.0541</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mnist</td>\n",
       "      <td>7603</td>\n",
       "      <td>100</td>\n",
       "      <td>9.2069</td>\n",
       "      <td>0.3367</td>\n",
       "      <td>0.3605</td>\n",
       "      <td>0.3741</td>\n",
       "      <td>0.1361</td>\n",
       "      <td>0.2721</td>\n",
       "      <td>0.432</td>\n",
       "      <td>0.3673</td>\n",
       "      <td>0.2653</td>\n",
       "      <td>0.3946</td>\n",
       "      <td>0.3878</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>musk</td>\n",
       "      <td>3062</td>\n",
       "      <td>166</td>\n",
       "      <td>3.1679</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.125</td>\n",
       "      <td>0.975</td>\n",
       "      <td>1</td>\n",
       "      <td>0.175</td>\n",
       "      <td>0.125</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>optdigits</td>\n",
       "      <td>5216</td>\n",
       "      <td>64</td>\n",
       "      <td>2.8758</td>\n",
       "      <td>0.0152</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0303</td>\n",
       "      <td>0.2121</td>\n",
       "      <td>0.0303</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0303</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>pendigits</td>\n",
       "      <td>6870</td>\n",
       "      <td>16</td>\n",
       "      <td>2.2707</td>\n",
       "      <td>0.0526</td>\n",
       "      <td>0.1579</td>\n",
       "      <td>0.0526</td>\n",
       "      <td>0.2632</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.0702</td>\n",
       "      <td>0.0702</td>\n",
       "      <td>0.0877</td>\n",
       "      <td>0.3158</td>\n",
       "      <td>0.3158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>pima</td>\n",
       "      <td>768</td>\n",
       "      <td>8</td>\n",
       "      <td>34.8958</td>\n",
       "      <td>0.5253</td>\n",
       "      <td>0.6061</td>\n",
       "      <td>0.4444</td>\n",
       "      <td>0.5354</td>\n",
       "      <td>0.5152</td>\n",
       "      <td>0.5859</td>\n",
       "      <td>0.4646</td>\n",
       "      <td>0.5152</td>\n",
       "      <td>0.5051</td>\n",
       "      <td>0.5354</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>satellite</td>\n",
       "      <td>6435</td>\n",
       "      <td>36</td>\n",
       "      <td>31.6395</td>\n",
       "      <td>0.3962</td>\n",
       "      <td>0.345</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.57</td>\n",
       "      <td>0.5825</td>\n",
       "      <td>0.4988</td>\n",
       "      <td>0.395</td>\n",
       "      <td>0.6762</td>\n",
       "      <td>0.5225</td>\n",
       "      <td>0.465</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>satimage-2</td>\n",
       "      <td>5803</td>\n",
       "      <td>36</td>\n",
       "      <td>1.2235</td>\n",
       "      <td>0.2333</td>\n",
       "      <td>0.6667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.8667</td>\n",
       "      <td>0.4333</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.6667</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0.7333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>shuttle</td>\n",
       "      <td>49097</td>\n",
       "      <td>9</td>\n",
       "      <td>7.1511</td>\n",
       "      <td>0.2003</td>\n",
       "      <td>0.2025</td>\n",
       "      <td>0.0257</td>\n",
       "      <td>0.9985</td>\n",
       "      <td>0.9596</td>\n",
       "      <td>0.212</td>\n",
       "      <td>0.1548</td>\n",
       "      <td>0.7395</td>\n",
       "      <td>0.956</td>\n",
       "      <td>0.9516</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>vertebral</td>\n",
       "      <td>240</td>\n",
       "      <td>6</td>\n",
       "      <td>12.5</td>\n",
       "      <td>0.2143</td>\n",
       "      <td>0</td>\n",
       "      <td>0.1429</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0714</td>\n",
       "      <td>0.1429</td>\n",
       "      <td>0.0714</td>\n",
       "      <td>0.0714</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>vowels</td>\n",
       "      <td>1456</td>\n",
       "      <td>12</td>\n",
       "      <td>3.4341</td>\n",
       "      <td>0.6316</td>\n",
       "      <td>0.1053</td>\n",
       "      <td>0.3684</td>\n",
       "      <td>0.1579</td>\n",
       "      <td>0.1579</td>\n",
       "      <td>0.4737</td>\n",
       "      <td>0.3684</td>\n",
       "      <td>0.1053</td>\n",
       "      <td>0.2632</td>\n",
       "      <td>0.1579</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>wbc</td>\n",
       "      <td>378</td>\n",
       "      <td>30</td>\n",
       "      <td>5.5556</td>\n",
       "      <td>0.375</td>\n",
       "      <td>0.375</td>\n",
       "      <td>0.375</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.375</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.375</td>\n",
       "      <td>0.375</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Data #Samples # Dimensions Outlier Perc    ABOD   CBLOF      FB  \\\n",
       "0  arrhythmia      452          274      14.6018  0.3571  0.4643  0.4643   \n",
       "0      cardio     1831           21       9.6122  0.1918  0.7945  0.1781   \n",
       "0       glass      214            9       4.2056    0.25    0.25    0.25   \n",
       "0  ionosphere      351           33      35.8974  0.8431   0.549  0.8039   \n",
       "0      letter     1600           32         6.25  0.4375  0.0312  0.4062   \n",
       "0      lympho      148           18       4.0541       0       1       1   \n",
       "0       mnist     7603          100       9.2069  0.3367  0.3605  0.3741   \n",
       "0        musk     3062          166       3.1679     0.1    0.65   0.125   \n",
       "0   optdigits     5216           64       2.8758  0.0152       0  0.0303   \n",
       "0   pendigits     6870           16       2.2707  0.0526  0.1579  0.0526   \n",
       "0        pima      768            8      34.8958  0.5253  0.6061  0.4444   \n",
       "0   satellite     6435           36      31.6395  0.3962   0.345     0.4   \n",
       "0  satimage-2     5803           36       1.2235  0.2333  0.6667  0.1667   \n",
       "0     shuttle    49097            9       7.1511  0.2003  0.2025  0.0257   \n",
       "0   vertebral      240            6         12.5  0.2143       0  0.1429   \n",
       "0      vowels     1456           12       3.4341  0.6316  0.1053  0.3684   \n",
       "0         wbc      378           30       5.5556   0.375   0.375   0.375   \n",
       "\n",
       "     HBOS IForest     KNN     LOF     MCD   OCSVM     PCA  \n",
       "0  0.5714  0.6071     0.5  0.4643  0.4286     0.5     0.5  \n",
       "0  0.4521  0.6027  0.3562  0.1507   0.411  0.5342  0.6849  \n",
       "0       0    0.25    0.25    0.25       0    0.25    0.25  \n",
       "0  0.3922  0.5882  0.8824  0.7843  0.8627  0.6863  0.5686  \n",
       "0  0.0938  0.0625  0.3125  0.3438  0.1875   0.125   0.125  \n",
       "0       1       1       1       1       1       1       1  \n",
       "0  0.1361  0.2721   0.432  0.3673  0.2653  0.3946  0.3878  \n",
       "0   0.975       1   0.175   0.125       1       1       1  \n",
       "0  0.2121  0.0303       0  0.0303       0       0       0  \n",
       "0  0.2632  0.3333  0.0702  0.0702  0.0877  0.3158  0.3158  \n",
       "0  0.5354  0.5152  0.5859  0.4646  0.5152  0.5051  0.5354  \n",
       "0    0.57  0.5825  0.4988   0.395  0.6762  0.5225   0.465  \n",
       "0     0.6  0.8667  0.4333  0.1667  0.6667     0.9  0.7333  \n",
       "0  0.9985  0.9596   0.212  0.1548  0.7395   0.956  0.9516  \n",
       "0       0       0  0.0714  0.1429  0.0714  0.0714       0  \n",
       "0  0.1579  0.1579  0.4737  0.3684  0.1053  0.2632  0.1579  \n",
       "0     0.5     0.5     0.5   0.375     0.5   0.375   0.375  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Precision @ n Performance')\n",
    "prn_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
